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METHOD AND APPARATUS FOR DOWNSCALING A DIGITAL COLOUR MATRIX 
IMAGE 

The present invention relates to a method and apparatus for downscaling a digital col- 
5 oured matrix image by selected ratios M2/M1 and N2/N1, in which the matrix image in- 
cludes Ni rows, each row including Mi pixels, so that the intensity values of the pixels 
form the matrix and the pixels of different colours form a selected format, and in which 
scaling is used to form an output matrix, of a size M2 x N2, the pixels corresponding to 
sub-groups of the original matrix, in such a way that M2<=Mi and N2<=Ni, and from the 
10 intensity values of which pixels the value of each output pixel of the output matrix is cal- 
culated. 

Camera sensors are used when using digital cameras to take individual images or a 
video image. The sensor image can use various image formats, for example RGB8:8:8. 

15 RGB5:6:5, YUV4:2:0, and a raw-Bayer image. When the image is displayed in the view- 
finder (VF), which usually has a lower resolution than the Image sensor, the image must 
be formed in the sensor and scaled to a suitable resolution for the display. Images can 
also be zoomed (a smaller image is delimited from the image and then scaled) to the 
viewfinder. In zooming, there must be many steps from the full Image size towards the 

20 larger zooming, so that the result of the zooming will appear to be continuous. When 
video images are coded, the resolution of the video image is also usually lower than the 
resolution of the sensor. Thus a similar scaling is also required for video. Camera sen- 
sors can also be used equally well in portable devices as in cameras. 

25 Image scaling or modifying the size of digital images has been exploited in several digi- 
tal image-processing applications. The basic method in image scaling is to arrange a 
two-dlmensionally sampled signal in a new sampling array. 

A few possibilities for carrying out image sub-sampling are known from the literature on 
30 signal and Image processing. The sampling of the signal is essential part in the theory 
of signal processing and has been widely covered in the literature. Basically, It is a mat- 
ter of preserving the spectrum of the Image below Nyquist frequency. 

The general sub-sampling methods Include an anti-alias filter and the re-creation of the 
35 image from the samples. The sampled data is often got using a linear combination of 
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sampled input data and a specific core. 

Sampling algorithms are often compromises between the complexity of the algorithm 
5 and the quality of the image obtained. 

The simplest form of re-sampling is the 'nearest neighbourhood' method. This does not 
use anti-alias filtering of the original data, but only selects the nearest samples for the 
new sampling array. The image obtained as a result is thus poor, due to the anti- 
10 aliasing effect. 

There are many methods for selecting the coefficients for the core of the anti-alias filter. 

US publication 6,205,245 discloses one method, in which a colour image is scaled di- 
15 rectly from the sensor's matrix, in such a way that a pixel group, of which one is always 
processed one at a time, and which conresponds to each pixel of the final image, is de- 
fined. In this method, intermediate pixels are Jumped over, thus losing original informa- 
tion. 

20 The present invention is intended to create a new high-quality sub-sampling method 
and an apparatus requiring small amount of memory for implementing the method. The 
characteristic features of the method according to the Invention are stated in the ac- 
companying Claim 1 while the features of the conresponding apparatus are stated In 
Claim 5. Correspondingly, the features of the software method according to the inven- 

25 tion are stated in Claim 14. The use of the method according to the invention requires 
only small amount of memory and the method is efficient in terms of calculation. The 
quality of the output image is, however, high. The method is particularly suitable for low- 
power devices such as cameras, as well as for camera and multimedia communicators. 
The small memory requirement is due to the fact that, in the scaling, one input image 

30 row, i.e. line, which is stored by summing only the amount of a single output line (row) 
into the line memory, as well as storing in a second line memory that part of the input 
image line, which is not contained in the output image pixel being processed. In one 
application, the pair of memories are alternated automatically by addressing the modulo 
of the index of the output row with the modulo 2-function, i.e. with the least significant bit 

35 of the integer (index). In one application, the memory required in scaling is implemented 
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in the processor (CPU) performing the scaling. This can be a separate DSP (digital sig- 
nal processor) - circuit component. One example of DSP processor architecture, \Nh\ch 
can be applied In context with the invention, is the multiprocessor ADSP-BF561 manu- 
factured by the Analog Device Inc. (USA) corporation. A second example referred to 
5 here is the RISC processor ARM9EJ-S (product specification: ARM DDI 0222B) of ARM 
Ltd., UK. In several two-processor applications, the processors are, however, located 
physically in different circuits/modules. 



10 



In the following, the invention is examined with reference to the accompanying figures, 
which show the method and apparatus of the invention. 



Figure 1 shows the scaling concepts 

Figure 2 shows an example of an output pixel 

Figure 3 shows a depiction of different types of pixels 

15 Figure 4 shows the structure of the memory required in the method 

Figure 5a shows the scaling algorithm as a flow diagram 

Figure 5b shows the use of DATA memory locations in a most difficult case 

Figure 6 shows the raw-Bayer image format and its scaling geometry 

Figure 7 shows one apparatus solution as a block diagram 

20 Figure 8 shows the scaling solution of Figure 7 at the circuit level 

Figure 9a shows the scaling stages of the raw-Bayer image format 

Figure 9b shows, as an alternative output form to that of Figure 9a, combined co- 
sited RGB, 

Figure 10a shows downscaling of the Bayer format to two different formats (co-sited 

25 RGB and Bayer), 

Figure 10b shows downscaling of the YUV 4:2:0 (H) format, 

Figure 1 0c shows downscaling of the YUV 4:2:2 (H) format, for an individual pixel, 

Figure 1 0d shows downscaling of the YUV 4:4:4 (H) format, 

Figure lOe shows downscaling of the YUV 4:4:4 format to a different format YUV 

30 4:2:2. 



35 



According to Figure 1, the input image (h) and the output matrix (I2) are placed on top of 
each other, so that each pixel of the output matrix is influenced by the original pixels. 
Scaling is a process, in which an output image with a lower resolution is formed from 
the input image. In order to simplify the depiction. Figures 1 - 5 mainly deal with the 
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scaling of a black-and-white image. The special features of scaling a coloured image 
will become apparent later. 

The image sizes are h: Mi x Ni and la: M2 x N2. The pixels are regarded more as areas 
5 than as points. The pixel sizes are set as Sxi x Syi for the input image and 8x2 x Sya for 
the output image. The image size is defined as L 'units', so that L = SxiMi = SX2M2 = 
(pixel size) x (number of pixels), so that 8x1/8x2 = M2/M1 = c x M2/C x Mi. 
The pixel sizes are then calculated as follows: 
8x1 = c X M2, and 
10 8x3 = c X Ml, 

c can be chosen freely: for example, c = 1, or c = 1/gcf(Mi,M2) (gcf: greatest common 
factor), or c = I/M1, or any c > 0. It is useful to choose the coefficient c in such a way 
that 8x2 is the power of two (1, 2, 4, ...), because divisions by 8x2 can then be easily 
performed in binary calculation (i.e. division becomes a transfer in binary integer calcu- 
15 lation). The pixel indices are marked as follows: (i = 0, 1, ... , (M1-I), j = 0, ... , (N1-I), k = 
0, ... ,(M2-1)andl = 0. ... ,(N2-1). 

The value of the output pixel is the weighted mean value of the values of the input pix- 
els that 'belong' to the area of the output pixel, Figure 1 . 

20 Out{kJ) = I W(iJ)In(i.j)\ IScaleFactor 

The scaling factor 8caIeFactor is calculated as follows: 

ScaleFactor = Z W{U j) =^.^ 

W(i.j) is the weighting coefficient. It is relative, depending on how many Input pixels 
there are inside an output pixel: 
25 W(i,j) = the surface area covered by the output pixel (k,l) inside (i, j) / the surface area of 
the input pixel (i,j). 

Figure 2 shows an example of this. The output pixel covers the input pixels: j = 0 ... jend, 
and i = 0 ... iend- The Indices (i=0,j=0) are used to depict the input pixels that initiate an 
30 arbitrary output pixel. Some weighting coefficients: 

W(0,0) = (Vstart X Wstart) / (Syi X 8X1) 

W(1.1) = 1 
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W(1.0) = Vstart/Syi ... etc. 

The following describes the application of the scaling method. 

In order to calculate the output pixel value, it is possible to use separate processing, i.e. 
5 the X direction can be calculated first, followed by the y direction: 

OutxQ) = [Wstart/SXi X ln(0,j) + Sum{ln(1J) ... In(iend-1 J)) + Wend/SXi X In(iendJ)] X SXi / SX2 
= [Wstart X ln(0,j) + SXi X Sum(ln(1 J) ... In(iend-I.j)) + Wend X ln(iend j)l / SX2. 

Out = [Vstart/Syi X Outx(O) + Sum(Outx(1) ... OutxGend-l) + Vend/Syi X OutxOend)! X Syi / Syz 

= [Vstart X OubC(O) + Syi X Sum(OutX(1) ... OutxGend-l)) + Vend X OutX(jend)] / Sy2. 

10 

This algorithm is based on online processing. One input pixel is processed at a time. 
The pixel data is summed in the line-memory structure. The weighting coefficients act- 
ing on the various indices can be calculated ready in one line memory (preferably the 
input line memory) in connection with processing, thus further reducing the number of 
15 calculation operations. The price for this is, of course, a slight increase in memory, but 
this Is repaid in the form of saved operations. Of course, two line memories will be re- 
quired, if the vertical and horizontal scaling coefficients are different. When the coeffi- 
cients are equal, only one (longer) line memory will be required. 

20 The input pixels have a different coefficient, depending on where they are located rela- 
tive to the output pixel. Thus, there are three types of pixel, see Figure 3. 

Full pixel/full pixel row: the Input pixel is entirely inside the output pixel. 
Final pixel/final pixel row: the input and output pixels have a common end edge. 
25 Part pixel/part pixel row: the area of the input pixel extends to the area of two or four 
output pixels. 

The memory structure is shown in Figure 4. It includes two parts, termed 'Data' and 
'Buffer'. The data structure includes four memory locations (Data[0/1][0/1]). The row 
30 memory structure includes two line memories (Buffer[0][0 ... M2-I], Buffer [1][0 ... M2- 
1]). The length of the line memories is the width of the output image. Separate memo- 
ries will always be required for the different colour components when the image is not 
read many times from, one of the large memories and processed only one component at 
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a time. In the case of a Bayer image, botli Gr and Gb are processed as tlieir own colour 
components and thus require their own memories. 

Because the sub-sampling method can be separated into parts, it is always possible to 
5 canry out vertical and horizontal scaling in separate processors (for example, in the 
camera sensor and the base-band processor). One-dimensional scaling does not re- 
quire a line-memory at all, if the scaling takes place in the direction in which the pixels 
come into scaling (generally in the horizontal direction). Only a data structure, such as 
DATA[0/1] is needed, when one-directional sub-sampling should be carried out. This 
10 data structure is required on rows or columns including all colour components. 

Depending on the type of pixel in question, the DATA and line-memory locations are 
filled. The relation of the pixel position to the output pixel (k, I) also determines how the 
memory structure is used (see the pseudocode later). 

15 

On the part-pixel line, both line memories Buffer[0][0 ... M2-I] and Buffer [1][0 ... M2-I] 
are in use. This means that the start of the following output line has already been proc- 
essed. In the part-pixel area, the data stnjctures Data[a][0] and Data[a][1] are summed, 
in which a can be 0 or 1 (full pixel row), or both 0 and 1 (part pixel row). 

20 

Figure 5a shows a flow diagram of the algorithm. We start from the first input row and 
from the first output row (i j,k.I=0, stage A). The type of pixel row is derived from the lo- 
cation of the pixel, stage B. The input row is processes according to its type (Stage C, 
D, or E). After the processing of the row, the next row always comes for processing, un- 

25 til all the rows have been processed. The part pixel row stage E, or the final (full) pixel 
row on the output row, stage D, terminate the output row. After a full row and a part pixel 
row, the next output row (1+1) is taken for processing, until the row that was processed 
was the last one G= N^-l), condition G, in which case the scaled image is ready. After 
this condition, the process either terminates or moves to the following output row, stage 

30 F. At the same time, the process moves to the processing of the following input row 
G=j+1), stage H. which is reached also from the processing of the full row in the middle 
of the output row, stage C. 
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Figure 5b illustrates the operation of the DATA memory locations in a most difficult 
case, i.e. in the case of a part pixel row. In this figure, the value of the output pixel of 
column k on row I is processed and in this case the input-Image pixels I from 61 - 64 
are used in the processing. The even-numbered indices (modulo 2) direct to the first 
5 memory location DATA[0][0]. When processing the previous output pixel (k-1), the part 
Intensity (upper broken line) of its last input-image pixel j=61 was stored. Here the 'full' 
vertical part-intensities (Xup) of the input pixels 62, 63 belonging to the same output 
pixel k, and this part intensity of the input pixel 64 are summed in it. The final part is 
stored in the memory location DATA[0][1] to form a 'seed' for the next pixel sum, Simul- 

10 taneously, in the processing of each pixel i, the Xdown part intensities were summed in 
the memory location DATA[1][0] and, in the case of the last one, the longitudinal part in- 
tensity in the memory location DATA[1][1], to form a seed for the pixel sum of the follow- 
ing row. After this, the contents of the memory locations DATA[0][0] and DATA[1][0] are 
summed to the elementary units BUFFER[0][k] and BUFFER[1][k] of the row memories. 

15 The first of these form the value Output[l][k] and is taken out. 

The following described the pseudocode for implementing scaling by software means. 

The output row (I) determines basically, which part of the line memory and the Data 
20 memory is used (Data[l mod 2] and Bufferll mod 2], see pseudocode). The number (k) 
of the output pixel determined the actual memory location (Data[l mod 2] [k mod 2] and 
Buffer[l mod 2] [k]). In this case, the term 'X mod 2' refers to X modulo 2, i.e. the re- 
mainder X when dividing by 2. The memory locations are automatically alternated by 
taking the least significant bit of the binary integer index as the address. 

25 

The following pseudocode depicts how the line-memory structure Is exploited. 
We use the following notations: 
30 Input pixel: lnput[j][i], 

Output pixel: Output[l][k], 



f 
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Weighting coefficients: 

Wend(k) = FRACTION [ (k+1) * M1/M2]: FRACTION is a fraction 

5 Vend(l) = FRACTION[ (1+1) * N1/N2] 
Ml, M2: number of pixels in the row 
Ni, N2 number of rows 
Sxi, Sx2, Syi, Sy2: size of pixels 
Scaling factor ScaleFactor (as above) 

10 Wstart[k+1] = 1 - WendM {= Wstart/Sx7 [0.1]), 
Vstart[l+1]= 1 -Vend[l]- 

For the type of pixel row. the value (= KA) is calculated for the statement ((1+1) x N1/N2- 
j), unliBss the row in question is the final row (index j = N1-I). If KA > 1, the row is a full 
row and in the case KA < 1 it is a part row. If KA = 1 , the row is the final row for the rele- 
15 vant output row. The memory recitation commands have been omitted. Each memory 
location is always reset, when data is forwarded. 

Processing of a full pixel row 

Full pixel 

20 Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + InputO][i] 
Final pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + lnput[j][i] 
Buffer[l mod 2][k] = Buffer[l mod 2] [k] + Data[l mod 2]Ik mod 2] 

25 

Part pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + lnput[j][i] x Wend[k] 

Data[l mod 2]I(k+1) mod 2] = lnput[j][i] x wstart[k+1] 

Buffer[l mod 2][k] = Buffer[l mod 2][k] + Data[l mod 2][k mod 2] 

30 

Processing of the final pixel row: 

Full pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + lnputD][i] 
Final pixel 
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Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + lnput[j][i] 
Bufferll mod 2][k] = BufferQ mod 2] ][k] + Data[l mod 2][k mod 2] 
Output[l][k] = Buffer[l mod 2][k] / ScaleFactor 

Part Pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + lnputD][i] x Wendtk] 
Data[l mod 2][(k+1) mod 2] = lnput|j][i] x wstart[k+1] 
Buffer[l mod 2][k] = Buffer[l mod 2][k] + Datall mod 2](k mod 2] 
Output[l]{k] = Buffer[l mod 2][k] / ScaleFactor 

Processing of a part pixel row: 

Xup = InpUtOjIi] X VendW 

Xdown = InputQlIi] xvstart[l+1] 

Full Pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + Xup 
Datal(l+1) mod 2][k mod 2] = Data[(l+1) mod 2][k mod 2] + Xdown 

Final pixel 

Data[l mod 2][k mod 2] = Data[l mod 2][k mod 2] + Xup 
Datal(l+1) mod 2][k mod 2] = Data[(l+1) mod 2][k mod 2] + Xdown 
Bufferp mod 2][k] = Buffer[l mod 2][k] + Data[l mod 2][k mod 2] 
Buffer[(l+1) mod 21[k] = Buffer[(l+1) mod 2][k] + Data[(l+1) mod 2][k mod 2] 
Output[l][k] = Buffer[l mod 2][kl / ScaleFactor 

Part pixel 

Data[l mod 2][k mod 2] = DatatI mod 2][k mod 2] + Xup x Wend[k] 
Data[l mod 2][(k+1) mod 2] = Xup x Wsiart[k+1] 

Datal(l+1) mod 2][k mod 2] = Data[(i+1) mod 2]tk mod 2] + Xdown x WenJk] 
Data[(l+1) mod 2][(k+1) mod 2] = Xdown xws,art[k+1] 
Buffer[l mod 2][k] = Buffer[l mod 2][k] + Data[l mod 2][k mod 2] 
Buffer[(l+1) mod 2][k] = Buffer[(l+1) mod 2][k] + Data[(i+1) mod 2][k mod 2] 
Output[l][k] = Bufferll mod 2][k] / ScaleFactor 
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BAYER IMAGE FORMAT 

Scaling can be perfonmed directly from a Bayer format (Figure 6). In a Bayer image, 
there is only one colour component in one pixel. The sampling frequency of the colour 
5 components (R, Gr, Gb. and B) is half of the sampling frequency of the pixels. Thus, the 
scaling factor for the colour components is tmce that of the scaling factor M2/M1 x N2/N1 
between the input and output Images, when scaling from a Bayer format to the co-sited 
RGB format. The scaling factor can be stated in the more general* form f x M2/M1 x 
N2/N1, In which f is the factor creating the format conversion. More specifically, the factor 
10 f = fw X fN. i.e. it too can be divided Into horizontal and vertical components. If the format 
does not change in scaling, then the value of f will be 1 (e.g., scaling from a Bayer for- 
mat to a Bayer format). 

The origin is marked in the figure with a circle in the original format and in the input pix- 
15 els of each colour. If, due to the offsets of the input image, pixel data does not exist for 
the entire area, then in that case it is advantageous to extrapolate slightly more the 
value of the previous pixel, so that there will be no need to alter the total scaling ratio in 
different ways for the edge pixels. Examples of the scaling of different formats are given 
below. 

20 

The offset of the output image can be implemented using simple counters or adjusting 
the different colour components to be the same. This means that the output image is 
offset relative to the input image. In practice, this means that part of the information of 
the edge pixels of the input image are left unused while correspondingly additional in- 
25 formation must be extrapolated at the opposite edge. 

The Invention is particularly suitable for hardware-based applications, one example of 
which is the application according to Figure 7. The camera module 10 is connected to a 
host system 22, which controls the display device 24 and the camera module. The cam- 
30 era module 10 includes particularly optics, i.e. a lens arrangement 11 (in practice sev- 
eral lenses), a sensor 12, an image-processing circuit 14, a scaling unit 16, and a con- 
troller component 20. The image-processing circuit 14 reads, in a known manner, the 
sensor 12, so that a high-speed data stream is created, which is led to the scaling com- 
ponent 16, from which the data stream of the scaled image is led to the host system 22. 
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In this case, the memory requirement is about two input-image lines. This means, for 
example, Cx2x160 words in a 160x120 matrix, in which C is the number of colour com- 
ponents (generally 3 - for RGB or YUV images). The length of the data word depends 
on the precision of the calculation and is, for example, 2 or 4 bytes. 

5 

At the circuit level, the structure of the scaler is in one (fully digital) application according 
to Figure 8. The scaler 16 includes an input component 161, a CPU 162. a memory 163, 
an output component 164, and a control component 167. all connected to an internal 
bus 165, This output component 164 feeds the scaled output pixel values (Output[l][k] to 
10 the host system 22. 

The apparatus can include a scaler component, in which there are separate processors 
(CPUs) for scaling in each dimension (for example, the aforementioned ADSP-BF561). 
The memory can be memory that is physically inside the processor. 

15 

In one application (scaling ratio less than 14). the apparatus includes memory for the 
scaling function for at most one Image sensor line for each colour component. 

The apparatus is highly suitable for a mobile terminal. 

20 

The following are various alternative implementations of the Invention. If there are two 
devices in the system (a camera module and the host system), the scaler can be Im- 
plemented in very many different places and in many different ways. 

1 . The sensor image Is processed and scaled by the module and is then sent to 
25 the host system (Figure 7). 

2. The sensor image is scaled and processed by the module and then sent to the 
host system. 

3. The sensor image is processed by the module and is then sent to the host sys- 
tem, in which it is scaled. 

30 4. The sensor Image is scaled by the module and then it is sent to the host system, 

in which it is processed. 

5. The sensor image is sent to the host system, in which it is scaled and proc- 
essed. 

e.The sensor image is sent to the host system, in which it is processed and scaled. 
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Scaling can always also be carried out In the display device, provided it lias a process- 
ing capacity. In addition, the image need not come directly from the sensor/camera 
module, but instead it can also be. read from the memory/memory card. The image can 
also be compressed (e.g, JPEG), in which case encoding is performed first, followed by 
5 scaling. These stages too (decoding and scaling) can be combined. In addition, situa- 
tions can also be found, in which the scaling can be carried out In two parts, in such a 
way that first of all the image is scaled in the horizontal direction by the camera module 
and then sent to the host system, in which it is scaled in the vertical direction and then 
processed. In addition, the scaler can be used in several (2) different stages, so that this 

10 small-memory scaler replaces the previous implementation of the scaler. In addition, the 
scaler can be used in such a way that a small image is rapidly formed on the display 
(using simple processing), and only then is the actual larger output image processed. If 
scaling takes place in the camera module, the host system must of course tell what size 
of image is required on the display. In addition, scaling may also be required in situa- 

15 tions, in which a video image is stored in the memory through a video encoder. In addi- 
tion, during storing it may be possible to view a viewfinder image on a differently sized 
display. Thus, situations can be found, in which the video encoder and not the display 
determines the output image. Similarly, there may be a need to use the same Idnd of 
scaler at two different ratios, either sequentially or in parallel, for different purposes. 

20 

According to Figure 9a, the colour components Gr and Gb, like R and B, are processed 
as totally separate components, which can then be finally combined. According to Fig- 
ure 9a, initially four separate components are processed, which form the elementary 
units of the scaled image directly in the Bayer format. When the output information is co- 

25 sited R, G, and B (Figure 9b), the two G components are combined into a single G 
component. (It is not essential to combine them when scaling to co-sited, but this is of- 
ten worth doing, in order to minimize the amount of data to be sent.) In order to keep the 
overall logic of the scaler simple, the G components are processed separately. Thus the 
component order is always retained in scaling (= the order of the incoming data is the 

30 same as the order of the outgoing data). It is important only to know that the question is 
of a Bayer as an input image. It should also be known whether the output image is 
Bayer or co-sited, so that the conrect offsets (and scaling factors) will be used. This con- 
cerns other colour formats as well, such as YUV and RGB. The order of the output 
components will only change if a switch is made to a different sub-sampling (e.g., from 
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YUV4:2:2 to YUV4:4:4), or a change, e.g., from an interleaved format to a progressive 
format takes place while scaling . 

Because scaling can perfectly well be performed separately (first in the x direction and 
5 then in the y direction), and because scaling can only be performed in one direction, it is 
sometimes clearer and simpler to consider the scaling factors only in one direction. 

In certain image formats (YUVx;x:x, Bayer RGB, co-sited RGB, etc., interleaved/planar 
formats) the pixel data is read from a data flow, a memory, or somewhere else, in a pre- 
10 defined order, or correspondingly the pixel data is written to a data flow, memory, or 
somewhere else in a predefined order. In Figures 10a - 10e, this data flow is marked 
'INPUT data order* or 'OUTPUT data order'. In the data order, the size of the pixel is of 
no importance, but for reasons of clarity the pixels are drawn in the figures according to 
the scaling ratio. 

15 

In scaling according to the patent application, it is, however, important to know the posi- 
tions and areas of the pixels (Figure 1 and Figure 2). The generally known image for- 
mats also define the positions and areas of the pixels and these are marked 'INPUT 
pixel positions' or 'OUTPUT pixel positions' in Figures 10a - lOe, The scaling geometry 
20 is intended to match the input and output pixels with each other (according to Figures 1 
and 2). 

The broken line on the left-hand side in the figures refers to the origin, point (0, 0). Gen- 
erally, the term origin refers to the left-hand upper comer of the image. In figures 10a - 
25 lOe, a few of the first pixels are shown, starting from the origin. 

In the following images Si = Sx^ or Si = Sy-,, and S2 = Sxz or Syz (as in Fig. 2) according 
to the situation. Si and S2 refer to the size of the pixels. 

30 Figure 10a shows the same Bayer-image scaling geometry as Figure 6. Due to the con- 
struction of a Bayer-image sensor, the colour components have their own well-defined 
position. Due to this, the colour components are moved suitably relative to the origin (in 
the image by the amount Si/2, S2/2, Sx/2, or Sy/2). so that the information of the colour 
components will also match each other. In this figure, the question is of Bayer to co- 
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sited RGB and Bayer-to-Bayer scaling. In this case, it sliould be noted that when an im- 
age is scaled to the Bayer format, the image obtained after scaling can be interpolated, 
by CFA interpolation, to the size indicated by the number of pixels. When scaling to co- 
sited format, CFA interpolation is not required, by size of the output image is halved 
5 from the number of pixels after scaling by finally combining the components. If the reso- 
lution of the Initial image is thus, e.g., 1600 x 1200 pixels and the scaling factor S1/S2 is 
0.5 (= 14) in both directions, then the size of the output image of Bayer scaling will be 
800 X 600 pixels, from which an output image of a size 800 x 600 can be interpolated. 
Correspondingly, when scaling to co-sited format, the size of the scaling output image 

10 will also be 800 x 600, from which a (4x)400 x 300-sized output image can be created. 
Correspondingly, if, in the co-sited format, the G-components are combined, then the 
size of the output image after scaling would be (3x)400 x 300 (i.e. 25 % fewer pixels 
than without combining). This leads to the doubling of the scaling factor referred to ear- 
lier. Thus the real scaling factor S2/S1 is actually 2 x (Sa/Si), that is, if this is compared 

15 using the ratios of the sizes of the output images created (i.e. for example, the scaling 
factor for a co-sited image is in practice, in both the x and the y direction 1 / scalefactor 
= 0.25 = 400 / 1600 = 300 / 1200). Thus, this does not affect the scaling factor and cal- 
culations during the actual scaling, but only the ratios of the output images. 

20 Figure 10b - c, e. In the formats YUV4:2:0 and YUV4:2:2, the sampling frequency of the 
U and V components is half the sampling frequency of the Y component, in which case 
the size of a U or V pixel will be two times (2Si) the size (Si) of a Y pixel. In the 
YUV4:1:1 format, the sampling frequency of a U or V component is one-quarter of the 
sampling frequency of a Y component, so that correspondingly the size of a U or V pixel 

25 will be fours time the size of a Y pixel. 

Figure 10b, In the YUV4:2:0 format, only either the U or the V component is stored on 
each line. Thus a YU line or YV line can be processed in the same way. The notation 
UA/ means that the pixel can be either a U pixel or a V pixel per line. 

30 

Figure 10c shows the processing of an individual pixel somewhere in the middle of the 
scaling of the image YUV 4:2:2 (H). The scaling YUV 4:2:2 -> YUV 4:2:2 vertical is the 
same as YUV 4:4:4 -> 4:2:2. 

In this case, the values of the output pixels are calculated as follows: 
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Yout = [WsTART.Y • Yin(1) + Si • Yin(2) + Wend y • Yin(3)]/S2 

Vout = [WsTART.v • Vin(1) + 2Si • Vin(2) + Wend.v • Vin(3)]/2S2 (and U as for V) 



Figure 10d shows YUV 4:4:4 -> YUV 4:4:4 scaling. (Co-sited RGB scaiing is the same 
5 as YUV 4:4:4, if R. G. and B are written in place of Y. U, and V.) 



Figure lOe shows YUV 4:4:4 -> YUV 4:2:2 scaling. 

A simultaneous sub-sampling variation and suitable scaling can be easily derived from 
10 the above examples. An example of this would be YUV 4:2:0 to YUV 4:4:4 conversion, 
in which scaling takes place by 0.5 in both the X and the Y direction. In that case, only 
the Y component need be scaled while the colour components remain unchanged. 

By using this method, the same advantages as in other image scaling can be obtained 
15 in scaling a Bayer Image. This are the correct use of the image information and the best 
possible filtering. In addition, Bayer scaling will substantially reduce the total calculation 
requirement and data traffic between processing. The method described to scale all 
components separately also brings the advantage that the precise order of the colour 
components need not be known in the scaler and thus, for example, image rotations, 
20 mirrorings, or sensor readings in different directions can be taken care of easily, without 
having to alter the scaler or its operation. In addition, when using different offsets and 
component settings, the scaler can be used for scaling very different kinds of colour- 
image information, and also for creating different kinds of output format. It can also be 
used to generate, for example, either Bayer or co-sited output images as required. 



